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FOREWORD 


This  document  describes  the  complexity  and  burgeoning  inter-  and/or  intraconnectivity  demands 
that  have  imposed  an  infrastructure  within  the  Naval  Warfare  Enterprise  that  includes  test  and 
evaluation  as  a  significant  aspect  of  each  Navy  Combat  System  Acquisition  Program  as  well  as 
support  for  deployed  units.  This  paper  will  provide  insight  into  the  current  state  of  the  Navy 
land-based  test  infrastructure  and  capability  and  how  it  is  exploited  for  large-scale  complex 
system  acquisition  support.  Strategic  options  for  adjustment  to  optimize  the  infrastructure, 
funding  paths,  process,  and  organizational  alignment  and  potentially  offer  a  more  cost-effective, 
integrated  approach  to  providing  certified  Navy  warfighting  capability  are  explored. 

This  document  has  been  reviewed  by  G.  Dale  Galyen,  Deputy  Department  Head,  Warfare 
Systems  Department. 


Approved  by: 


DONALD  L.  BURNETT,  Head 
Warfare  Systems  Department 
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GLOSSARY 


Acronym 

Definition 

ASN(RDA) 

Assistant  Secretary  of  the  Navy  (Research,  Development,  and  Acquisition) 

COMNAVSEA 

Commander,  Naval  Sea  Systems  Command 

CONUS 

Continental  United  States 

GFE 

Government-Furnished  Equipment 

LBTS 

Land-Based  Test  Site 

LSI 

Large-Scale  Integration 

NAVAIR 

Naval  Air  Systems  Command 

NAVSEA 

Naval  Sea  Systems  Command 

NGS 

Next-Generation  Simulation 

NSWCDD 

Naval  Surface  Warfare  Center,  Dahlgren  Division 

NSWCPHD 

Naval  Surface  Warfare  Center,  Port  Hueneme  Division 

PEO  IWS 

Program  Executive  Office  Integrated  Warfare  Systems 

PM 

Program  Manager 

SES 

Senior  Executive  Service 

SoS 

System  of  Systems 

T&E 

Test  and  Evaluation 

TC 

Technical  Capability 

TEMP 

Test  Engineering  Management  Plan 
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1.0  SCOPE 


1.1  Introduction 

Key  to  the  success  of  almost  any  surface  ship  combat  and/or  warfare  system  acquisition  program, 
test  and  evaluation  (T&E)  is  a  complex  process  of  requirements  evaluation,  Test  Engineering 
Management  Plans  (TEMPs)  development,  testing  development,  scheduling,  execution,  analysis, 
and  certification.  Each  component,  element,  subsystem,  and  system  within  a  ship  is  tested 
extensively  to  ensure  that  the  ship’s  systems  meet  mission  requirements,  are  safe  to  operate,  and 
are  effective.  A  building  block  approach  is  used  by  each  acquisition  program,  which  tailors  the 
test  process  to  the  program  budget  and  direction  of  each  Program  Manager  (PM).  The  result  is  a 
different  test  process  for  almost  every  program,  depending  on  the  requirements,  the  platfonn,  and 
which  organization  leads  the  test  process.  This  approach  allows  each  acquisition  program  to 
develop  infrastructure  that  is  finely  tuned  to  the  respective  program’s  requirements.  However,  it 
can  also  result  in  duplicate  organizations,  infrastructures,  and  investments  that,  while  creating 
efficiencies  for  individual  programs,  can  create  unnecessary  expenditure  of  the  overall  Surface 
Navy  combat  system  integration  and  test  budget. 

Determining  T&E  cost  is  a  hugely  complex  issue,  and  it  is  worth  noting  that  testing  in  a  land- 
based  test  environment  can  cost  eight  to  ten  times  less  than  it  costs  the  Navy  to  conduct  the  same 
test  at  sea.  Empirical  data  indicates  that  most  relatively  complex  acquisition  programs  expend 
nearly  30  percent  of  their  budgets  for  integration  and  test  purposes.  Thus,  presenting  options, 
opportunities,  and  a  strategy  for  cost-effective  Surface  Navy  combat  system  integration  and 
testing  within  recognizably  constrained  budgets  is  worth  pursuing.  The  combination  of  people, 
processes,  and  facilities  together  fonns  the  basic  T&E  capabilities  across  the  Warfare  Centers 
and  test  sites  for  the  Naval  Sea  Systems  Command  (NAVSEA)  and  the  Assistant  Secretary  of  the 
Navy  (Research,  Development,  and  Acquisition)  [ASN(RDA)].  Integration  of  systems  requires 
unique  expertise,  and  moving  from  a  loose  collection  of  systems  engineering  and  test 
engineering  talent  to  cohesive  activity,  alignment,  and  unity  of  effort  is  a  real  challenge.  Shifting 
from  industry  leadership  to  government  leadership  of  large-scale  integration  and  testing  is  also  a 
difficult  transition  to  make  because  so  much  govermnent  expertise  has  shifted  to  industry  due  to 
the  approach  taken  through  acquisition  reform. 

1.1.1  Background 

Following  the  Reagan  era  budget  plus-ups  toward  the  end  of  the  Cold  War,  the  United  States 
began  efforts  to  claim  the  new  “Peace  Dividend.”  Since  1985,  the  budgetary  high-water  mark  of 
the  Cold  War,  funding  for  defense  has  declined  and  the  “Peace  Dividend”  eroded.  Since  then, 
with  the  cost  of  the  Global  War  on  Terror  and  other  budget  issues,  the  test  community  within  the 
Surface  Navy  has  seen  a  steady  decline  in  funding  and  a  push  for  continued  cost  avoidance  by 
increasing  the  role  of  industry  in  the  acquisition,  systems  integration,  and  test  processes.  The 
Surface  Navy  test  community  has  struggled  to  accomplish  the  necessary  task  of  performance 
characterization  and  conduct  testing  sufficient  to  provide  inputs  to  the  certification  process  for 
Surface  Navy  systems  and,  more  precisely,  combat  systems,  despite  this  funding  decline. 
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With  the  looming  development  of  CVN  78,  DDG  1000,  the  Littoral  Combat  Ship,  Amphibious 
Transport  Dock,  Aegis  Modernization,  Aegis  Ballistic  Missile  Defense,  and  other  requirements 
facing  the  integration  and  test  community,  finding  a  means  to  cost-effectively  integrate  test 
systems  for  deployment  and  mission  execution  is  a  critical  component  of  each  acquisition 
program  budget. 

1.2  The  Building  Blocks 

Surface  Navy  combat  systems  are  complex  entities  designed  and  built  in  stages  of  increasing 
complexity  and  intra/intersystem  connectivity,  communication,  dependency,  and  operation.  The 
integration  and  test  process  applies  a  similar  building  block  approach,  with  hardware  and 
software  components  tested  as  they  are  developed.  As  they  are  grouped  within  functions  for 
system  performance,  they  become  elements,  subsystems,  systems,  and  a  system  of  systems 
(SoS).  For  shipboard  installation  and  test,  there  is  a  formal  seven-stage  process  as  presented  in 
DoD-STD-2106  that  reflects  a  similar  building  block  approach  (Reference  1).  This  approach 
(albeit  in  various  forms)  surfaces  at  the  factory  or  with  the  developer,  in  land-based  test  sites 
(LBTSs),  and  finally  aboard  ship.  Software  testing  for  Navy  combat  systems  holds  to  a  similar 
disciplined  building  block  approach  with  tailoring  by  each  organization  involved.  To  some 
degree,  migration  has  occurred  from  the  waterfall  model  for  software  systems  to  a  spiral 
model  for  development  and  deployment  (Reference  2).  However,  the  spiral  model  applies 
engineering  within  the  spirals,  such  that  the  building  block  approach  is  nevertheless  employed 
with  a  characteristic  “V”  shape  (see  Figure  1)  to  the  overall  process  from  concept,  through 
development,  to  deployment. 


Figure  1.  Systems  Engineering  V-Chart 


Waterfall  model  derived  from  the  cascading  effect  of  movement  from  one  phase  to  the  next,  where  each  phase  is 
generally  well  defined  and  separate. 

Spiral  model  combines  elements  of  design  and  development  stages. 
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The  V-Chart,  as  it  is  referred  to  in  the  systems  engineering  community,  rests  upon  a  foundation 
of  disciplined  testing  designed  to  assure  expected  performance.  Most  testing  stems  from  the 
systems  engineering  discipline  and,  in  fact,  many  consider  it  a  subspecialty.  However,  it  should 
be  noted  that,  from  the  competency  perspective,  T&E  stands  shoulder  to  shoulder  with  other 
competencies  in  the  Competency- Aligned  Organization  and,  at  NAVSEA,  is  covered  under  the 
cognizance  of  the  same  Senior  Executive  Service  (SES)  leader  as  the  Systems  Engineering 
Competency. 

T&E  execution  has  seven  basic  steps:  (1)  identifying  system  perfonnance  expectations  and  the 
test  requirements,  (2)  planning,  (3)  conducting  pretest  analysis,  (4)  tailoring  the  test  process, 

(5)  scheduling  and  executing  the  test,  (6)  conducting  posttest  evaluation  (analysis),  and 
(7)  reporting.  As  stated  previously,  these  steps  are  accomplished  according  to  each  program’s 
requirements  and  are  therefore  targets  of  opportunity  for  competency  development;  movement 
toward  more  common,  more  useable,  and  effective  tool  sets;  standardized  processes;  consistent 
and  persistent  teams;  and  formal  (and  possibly  accredited)  test  discipline. 

Testing  is  an  essential  feedback  loop  that  provides  assurance  that  the  system  or  SoS  meets 
capability  expectations  and  mission  requirements.  A  significant  portion  of  the  test  process  is  or 
can  be  conducted  at  LBTS  facilities  throughout  the  United  States.  These  sites  have  grown  in  size 
and  complexity  as  situations  have  merited,  as  requirements  have  dictated,  and  as  budgets  have 
allowed. 

1.3  Current  State 

Each  LBTS  facility  was  developed  to  address  requirements  within  a  particular  domain,  area  of 
responsibility,  or  technical  capability  (TC).  A  TC  is  defined  as  “representing  the  blending  of 
intellectual  and  physical  assets  provided  by  a  cadre  of  technical  people  with  knowledge,  skill, 
experience,  and  requisite  facilities  and  equipment  that  yield  the  ability  to  deliver  technical 
products.”  (See  Reference  3.)  For  instance,  the  Naval  Surface  Warfare  Center,  Port  Hueneme 
Division  (NSWCPHD),  lists  nine  TCs,  each  of  which  includes  T&E  in  the  title  and  in  the 
detailed  description  as  well.  As  it  turns  out,  there  is  overlap  between  the  NSWCPHD  TCs  and 
those  of  other  Warfare  Centers  (see  Figure  2),  such  as  Naval  Surface  Warfare  Center,  Dahlgren 
Division  (NSWCDD);  but  the  overlap  diminishes  with  an  understanding  of  each  TC’s 
application.  Some  examples  are  PH02  Surface  Combat  Systems  In-Service  Engineering  T&E, 
and  Integrated  Logistics  Support  for  NSWCPHD;  DD  05  Surface  Combat  Systems  Engineering 
and  Integration  Research,  Development,  Test,  and  Evaluation  for  NSWCDD;  and  DD  35 
Integrated  Surface  Combat  Control  and  Systems  Support  for  Dam  Neck. 

The  essence  of  the  difference  in  this  case  is  that  NSWCPHD  leads  the  at-sea  test  efforts,  while 
NSWCDD  and  Dam  Neck  lead  land-based  test  efforts  in  their  respective  domains.  As  can  be 
seen  from  the  example,  some  overlap  provides  a  mutually  supportive  and  beneficial  aspect 
embedded  in  the  warfare  system  TCs.  Over  time,  however,  the  same  apparent  overlap  has 
contributed  to  establishment  of  infrastructure  that,  if  left  to  individual  PMs  to  decide,  will  grow 
to  meet  individual  program  needs  but  won’t  necessarily  be  harnessed  collaboratively  to  meet  the 
requirements  of  the  larger  Navy  budget  constraints.  One  approach  to  rectify  the  drift  toward 
independent,  overlapping,  and  duplicative  capabilities  is  to  develop  partnerships  across  the 
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Warfare  Centers  that  leverage  existing  Warfare  Center  strengths  rather  then  developing  or 
creating  further  duplication. 


Figure  2.  Naval  Surface  Warfare  Center  Chain/Construct 


1.4  Opportunities 

There  are  many  significant  areas  in  the  Surface  Navy  combat  system  test  domain  that  together 
form  targets  of  opportunity  for  adjustment  to  achieve  greater  collaboration  across  the  Warfare 
Center  and  the  activities,*  and  achieve  significant  measures  of  cost  avoidance  and  cost  savings** 
for  the  Navy.  Factors  include  the  following: 

1 .  Surface  Navy  combat  system  T&E  requirements  are  significant  budget  drivers. 

This  significant  budget  driver  surfaces  for  many  reasons.  Significant  among  them  are 
the  high  cost  of  new  facilities  to  meet  new  requirements,  the  inefficiency  of  distributed 
investments  to  achieve  similar  goals  by  different  programs'  and  sponsors/  and  the 
overlap  of  government  and  industry  roles  and  infrastructure  that  drives  investment 
duplication.  An  example  of  infrastructure  is  the  costly  development  of  a 
simulation/stimulation  integration  and  test  environment  for  each  new  program. 

2.  There  is  currently  no  mechanism  for  singular  task  generation  across  Warfare  Centers  to 
address  test  requirements. 

Due  to  a  number  of  factors,  each  Warfare  Center  is  independently  funded  to 
accomplish  similar  tasking  across  different  programs  and  sponsor  requirements.  Warfare 
Centers  are  working  capital  fund  organizations.  In  essence,  that  means  that  when  a 
requirement  exists  that  a  Warfare  Center  can  fulfill  and  a  program  or  sponsor  is  willing  to 
fund,  the  center  is  paid.  That  is  opposed  to  mission  funding,  which  refers  to  being  a  part 
of  a  line  item  in  the  President’s  budget.  Other  influences  include  the  variation  of 


*  “Activity”  or  “activities”  sometimes  refers  to  sites  that  do  not  fall  within  the  Warfare  Center  construct. 

**  “Cost  avoidance”  is  akin  to  efficiency  in  that  it  refers  to  the  ability  to  do  more  or  the  same  for  less  money,  while 
“cost  savings”  addresses  money  returned  to  the  sponsor. 

1  “Programs”  in  this  context  refers  to  acquisition  programs  such  as  Aegis  or  Littoral  Combat  Ship. 

*  “Sponsors”  are  organizations  such  as  NAVSEA  05  that  have  requirements  that  must  be  funded  to  fulfill  but  do  not 
“own”  programs. 
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technical  capability  (which  incorporates  overlap),  personal  preference  or  bias,  and 
legitimate  program  requirements  that  must  be  met. 

3.  There  are  overlapping  site-,  Warfare  Center-,  industry-,  and  sponsor-supported 
integration  and  test  efforts  and  capabilities. 

The  last  target  of  opportunity  for  achieving  greater  efficiencies  and  effectiveness  is 
that  there  are  duplicative  investments  across  sites,  Warfare  Centers,  industry,  and 
sponsor-supported  integration  and  test  efforts  and  capabilities.  This  can  lead  to  the 
instantiation  of  redundancies  geared  toward  optimizing  for  legitimate  program 
requirements,  such  as  meeting  demand  versus  capacity  realities.  This  duplication  of  effort 
can  also  be  a  function  of  the  constructs  of  the  acquisition  process  that  does  not  easily 
provide  for  overarching  mechanisms  for  bigger  Navy  support  structures  as  opposed  to 
individual  program  requirements.  To  achieve  cost  avoidance  and  improvement  in 
competency  (Reference  4),  capability  (Reference  5),  and  effectiveness  and 
instantiating  an  independent  structure  for  combat  systems  integration  and  test 
requirements,  an  integrated  test  approach  can  be  stood  up. 

1.5  Desired  (Future)  State 

The  issue  for  attempting  any  alignment  of  effort  across  the  Warfare  Centers  is  that  each  center 
receives  funding  and  direction  from  a  myriad  of  customers,  such  as  NAVSEA  05  and  Program 
Executive  Office  Integrated  Warfare  Systems  (PEO  IWS),  to  accomplish  similar  work.  Thus, 
leadership,  cost  sharing,  and  control  surface  quickly  as  significant  issues  to  be  overcome  in  any 
associated  alignment  strategy.  However,  a  vision  can  be  achieved  for  unified  and  aligned  efforts 
(see  Figure  3),  and  the  many  Warfare  Centers  can  be  bent  to  operate  as  a  singular  integration  and 
test  enterprise  with  respect  to  Surface  Navy  combat  systems.  Such  a  state  would  account  for 
alignment,  from  the  bottom  to  the  top,  of  the  integration,  test,  and  evaluation  processes  as 
illustrated  in  Figure  4. 

1.5.1  Leadership 

In  order  to  achieve  the  vision  of  an  integrated  test  approach  across  the  Warfare  Centers,  we  may 
need  to  lessen  the  need  for  individual  program  management  to  control  every  aspect  of  the 
infrastructure  and  process  associated  with  that  program’s  success;  no  small  matter  to  consider.  It 
is  also  important  that  the  Warfare  Centers  and  other  integration  and  test  sites  operate 
collaboratively  rather  than  as  individual  organizations  looking  out  for  each  program,  without 
consideration  for  the  cost  to  other  programs  for  individual  approaches  to  the  Surface  Navy 
combat  system  test  process.  Responsibility  could  be  transferred  to  the  Integrated  Test  Team 
leadership  for  effecting  optimum  test  processes  across  acquisition  programs  and  for  optimizing 
to  achieve  maximum  cost  effectiveness  as  well.  With  an  eye  toward  meeting  multiple  rather  than 
singular  stakeholder  needs,  an  integrated  test  team  can  provide  the  linkage  between  capability, 
budget,  and  test  requirements,  as  shown  in  Figure  5.  However,  this  is  a  paradigm  shift  for 


The  state  of  having  attributes,  expertise,  and  knowledge  that  are  “adequate  for  the  stated  purpose”  refers  to 
individual  competence. 

'  The  cumulative  result  of  people,  their  individual  competences,  processes,  and  facilities  leveraged  to  achieve  the 
desired  outcomes. 
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NAVSEA  and  ASN(RDA),  and  a  commensurate  cultural  shift  must  occur  as  well,  leveraging 
both  top-down  and  bottom-up  leadership  to  achieve  success. 
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Figure  3.  Vision  for  an  Integrated  Test  Approach 
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Figure  4.  Overall  Aligned  Test  Approach 
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Integrated  Priorities,  Cross  Functional  Investment  Strategy,  Cross  Programs  &  Domain  T&E  Enablers 
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Figure  5.  Collaborative  Leadership  Applied  to  Support  the  Integrated  Test  Concept 


1.5.2  Constraints 

Success  may  require  that  individual  program  sponsors  relinquish  some  measure  of  control.  It  also 
requires  sufficient  budget;  but  what  constitutes  sufficiency?  Adaptability  to  constrained  budgets 
is  a  fact  of  life  given  the  level  of  funding  available  to  support  Surface  Navy  combat  system 
integration  and  test  requirements.  Perhaps  though  there  is  sufficient  budget  to  accomplish  the 
required  tasks.  What  is  lacking  is  sufficient  control  over  application  of  that  budget  to  optimize 
necessary  activity.  That  is,  consider  whether  organizational  tasking  might  enable  facility 
optimization  as  opposed  to  programmatic  tasking  for  each  program’s  integration  and  test 
objectives. 

If  each  program  developed  its  TEMP  but  leveraged  the  Integrated  Test  Team  to  do  so  and 
allocated  a  sufficient  percentage  of  funding  to  support  the  integration  and  test  process,  a  cross¬ 
organization,  cross-program,  cost-effective  test  strategy  could  be  developed.  Another  approach 
might  be  to  fund  testing  independent  of  programs.  That  is,  testing  could  become  a  Surface 
Navy/NAVSEA  budget  line  item.  This  might  prove  untenable  since  each  program  still  requires 
some  level  of  testing  to  prove  its  components  or  systems  meet  quality  standards  and  warfighter 
or  end  user  requirements,  at  least  through  the  development  cycle.  However,  a  single  sponsor 
could  accommodate  both. 
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2.0  STRATEGY 


2.1  Options 

One  approach  to  achieving  the  necessary  insight  into  combat  systems  performance  at  reduced 
cost  and  overcoming  the  duplicate  investment  issue  inherent  in  the  overlap  of  Warfare  Center-, 
industry-,  and  sponsor-supported  T&E  efforts  and  capabilities  is  to  realign  and  refocus  the 
Surface  Navy  test  community.  It  is  possible  that  the  number  of  systems  fielded  at  industry  and 
government  sites  can  be  reduced  by  reemphasizing  the  government  role  in  the  certification 
process.  That  is,  reduce  the  number  of  Navy-funded  systems  at  industry  sites  to  that  needed  for 
singular  interface  testing  while  still  providing  for  industry  partners’  development  laboratory 
requirements.  However,  there  should  then  be  an  integration-focused  site  that  industry  can  use 
when  system  development  requires  multiple  system-level  interface  testing.  This  would  be  the 
simulation/stimulation-driven  environment  for  integration  and  testing  in  a  controlled 
environment,  using  reduced  levels  of  tactical  equipment  and  heavier  use  of  commercial  or 
functional  equivalent  systems  built  at  significantly  lower  costs  than  those  of  tactical  suites. 
Figure  6  reflects  an  adjustment  in  site  focus  and  test  approach  that  can  address  significant  cost 
avoidance  in  the  integration  and  test  process. 


Industry 

-  Detailed  Development  Requirements 

-  Detailed  Engineering  Requirements 
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-Singular  Interface  Testing 
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Figure  6.  Aligned  Integration  and  Test  Leveraging  Government  Integration 


Finally,  the  Navy  could  build  one  full-up  tactical  suite  to  support  final  integration  and 
configuration  testing.  To  streamline  system  integration  and  startup  at  the  respective  sites,  the 
Navy  could  also  provide  one  simulation/stimulation  environment  designed  to  provide  all  the 
external  interfaces  with  which  a  combat  system  expects  to  interface,  as  government-furnished 


That  portion  of  the  test  environment  that  supports  controlled  testing  where  live  components  are  not  available 
(simulated)  and  stimulation  that  drives  the  system  under  test  to  respond. 
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equipment  (GFE)  to  industry  as  well  as  to  each  government  site.  A  government-funded 
simulation/stimulation  environment  could  reduce  the  cost  of  building  a  unique  test  environment 
at  a  site  as  the  suite  grows  in  size  and  complexity;  the  net  result  being  deployment  of  fewer 
tactical  suites,  each  of  which  takes  a  significant  portion  of  a  program  budget.  Compounded  with 
the  savings  from  the  government-funded  simulation/stimulation  environment,  this  approach  can 
bring  about  tremendous  cost  savings. 

As  an  example,  a  representative  GFE  environment  known  as  Next-Generation  Simulation  (NGS) 
costs  about  $300K,  where  simulation  development  individually  can  and  has  cost  millions.  One 
program  in  execution  today  estimates  dedicating  more  than  22,000  labor  hours  to  developing  a 
simulation/stimulation  capability  for  just  one  integration  and  test  requirement  interface  at  the 
industry  development  site.  Functionally  equivalent  systems  have  been  utilized  at  the  Surface 
Combat  System  Center  in  Wallops  Island,  Virginia,  which  have  resulted  in  excess  of  $500K 
savings  per  cabinet. 

2.2  Implementation 

To  effect  such  an  aligned  approach  requires  leadership  support  from  the  sponsor,  Office  of  the 
Chief  of  Naval  Operations,  from  the  PEO  major  PMs,  as  well  as  from  NAVSEA  and 
ASN(RDA).  The  Secretary  of  the  Navy;  Commander,  NAVSEA  (COMNAVSEA);  and 
Congress  agree  that  there  should  be  renewed  emphasis  on  government  leadership  of  large-scale 
integration  (LSI)  efforts.  This  approach  aligns  with  that  idea,  but  not  all  parties  agree  up  and 
down  the  chain  of  command  nor  does  industry  relish  the  loss  of  such  a  lucrative  and  profitable 
business  base. 

At  a  recent  all-hands  brief,  COMNAVSEA  presented  a  chart  that  illustrated  the  government 
infrastructure  size  that  would  be  leveraged  to  engineer  LSI  solutions  for  the  Navy.  This 
infrastructure,  the  Warfare  Center  Engineering  community,  a  portion  of  which  is  reflected  in 
Figure  2,  is  second  in  size  only  to  the  largest  defense  industry  conglomerates  but  operates  at 
significantly  less  cost  than  its  industry  contemporaries.  Moving  to  more  government-led 
integration  and  test  provides  significant  motivation  to  effect  alignment.  However,  true  success 
will  not  occur  until  the  bottom-up  approach  underway  across  the  Warfare  Center  construct  meets 
top-down  direction  from  appropriate  senior  Navy  leadership. 

2.3  Execution 

Several  initiatives  have  been  developing  momentum  to  achieve  the  outcomes  projected  in  an 
aligned  Surface  Navy  test  strategy:  moving  toward  common  data  analysis  tools  across  Warfare 
Centers;  consolidating  test  efforts  across  different  levels  (subsystem,  element,  and  system)  so 
that  less  redundant  testing  is  needed;  and  having  NGS  development  architected  to  provide  all 
external  interfaces  required  for  combat  system  testing  at  a  reduced  cost.  At  the  same  time,  SES- 
and  Flag-level  leadership  needs  to  develop  and  implement  motivational  factors  to  achieve 
success.  Rewarding  cost  reductions  over  some  significant  size  while  achieving  expected  results 
with  a  percentage  of  the  savings  or  avoided  costs  can  be  a  significant  motivator  across  the  Navy. 
Implementing  training  and  education  to  further  acculturate  the  PM,  NAVSEA,  and  ASN(RDA) 
work  force  would  help  institutionalize  the  continuous  improvement  and  streamlining  vision  until 
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it  becomes  a  natural  part  of  everyday  efforts.  Aligning  required  competencies  to  meet  the 
demand  signals  expected  from  such  an  approach  is  also  a  necessary  element  for  success. 
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3.0  RISK 


3.1  Risk  Impact 

Considering  both  the  probability  and  the  impact  of  risks  inherent  in  developing  an  overarching, 
consistent,  and  program-independent  integrated  test  team  approach  requires  keen  insight  into  the 
test  process.  While  Naval  Air  Systems  Command  (NAVAIR)  has  had  clarity  in  its  T&E  structure 
for  some  time,  the  NAVSEA  organization  is  spread  out  to  address  testing  at  a  different  level  of 
complexity.  NAVAIR  conducts  testing  of  aircraft,  complex  units  in  and  of  themselves;  but  the 
Surface  Navy  combat  system  level  of  complexity,  integrating  across  multiple  systems  and 
multiple  warfare  areas  and  missions  and  combining  complex  capabilities  while  allowing 
flexibility  and  agility  of  response,  has  pushed  NAVSEA  to  distribute  its  test  capability  by 
domain  rather  than  function.  Migrating  to  a  competency-aligned  organization,  with  a  robust  and 
focused  test  team  distributed  across  the  nation  while  maintaining  centralized  control,  is  no  easy 
task. 

Risks  include  growing  or  favoring  one  organization  over  another  as  leadership  shifts;  aggregate 
costs  for  test  capability  (people,  process,  and  facilities)  become  bundled  and  therefore  large 
enough  to  draw  attention  as  targets  of  opportunity  for  reduction;  and  an  inability  to  accurately 
identify  the  full  extent  of  test  costs  on  a  per  program  or  Warfare  Center-by-Warfare  Center  basis 
occurs.  The  complexity  of  shifting  from  a  distributed  or  federated  approach  under  control  of 
individual  programs  to  a  different  approach  also  has  risk.  The  shift  alone  could  cause  missed 
schedules  or  prove  testing  inadequate  if  not  appropriately  funded,  tooled,  and  given  solid 
leadership.  But  each  of  these  risks,  even  with  their  commensurate  impacts,  could  be  overcome 
given  solid  leadership  and  commitment  toward  such  a  shift  in  strategy.  It  should  be  noted  that  the 
Navy  often  operates  through  individual  interpretation  of  a  Commander’s  intent,  solid  leadership 
that  interprets  orders  and  intentions,  and  executes  on  its  own  at  every  level.  While  this  can  at 
times  be  considered  strength,  in  the  case  of  movement  toward  an  integrated  test  team  across 
distributed  Warfare  Centers,  the  individualist  approach  could  well  prove  catastrophic  and  has 
already  proven  expensive.  There  are,  however,  mechanisms  for  managing  these  risks. 

3.2  Risk  Management 

If  consensus  were  developed  top  down  as  well  as  bottom  up  and  a  shift  to  an  integrated  testing 
were  to  be  held  in  abeyance  until  discordant  requests  poured  in  as  the  concept  was  advertised, 
then  the  risk  of  splintering  as  the  shift  occurs  can  be  diminished.  Another  approach  is  strong  top- 
down  leadership  combined  with  consensus  and  bottom-up  leadership,  which  might  well  prove 
successful;  but  leadership  at  the  top  changes  often.  There  needs  to  be  some  institutionalizing  of 
the  integrated  test  approach,  with  bridges  burned  to  prevent  falling  back  to  how  we  used  to  do 
business;  there  might  then  be  a  greater  opportunity  to  manage  the  risk.  This  is  true  for  many  of 
the  risks  mentioned:  an  aggregate  budget,  integrating  across  multiple  programs,  even  the  shift  to 
centralized  direction. 
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Providing  customer  service  agreements  that  link  organizations  through  tasking  and  response  to 
sponsors  and  PMs  would  help  to  force  collaboration  across  the  construct  to  greater  and  greater 
levels  until  they  operate  seamlessly,  almost  indistinguishable,  one  from  the  other,  the 
distinguishing  element  being  primarily  domain  expertise.  Single  tasking  that  includes  or  even 
requires  multiple  organizations  to  achieve  expected  outcomes,  enforced  within  funding 
documents,  brings  with  it  the  force  of  a  contract.  When  that  occurs,  the  Warfare  Centers  may 
quickly  join  forces  to  achieve  success. 

The  potential  gain  is  worth  the  risks,  but  prudent  selection  of  both  the  Enterprise  Leadership 
Group  participants  at  the  certification  panels  and  knowledgeable  leaders  in  the  test  domain  from 
around  the  country  to  lead  and  direct  the  Integrated  Test  Team  can  make  the  shift  to  integrated 
test  under  centralized  leadership  of  distributed  test  capabilities  a  success. 
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4.0  CONCLUSION 


4.1  Measuring  Results 

Combat  system  and  platform-level  testing  costs  for  the  Navy  for  each  ship,  system,  subsystem, 
and  component  are  nearly  astronomical.  To  baseline  costs  using  today’s  approaches  and  then 
watch  the  cost  of  testing  over  time  as  shifts  toward  an  aligned  test  strategy  occur  is  one  measure 
of  effectiveness.  The  size  of  the  test  organization  capability  (Reference  6),  the  number  of 
resources  (people),  and  the  proliferation  of  common  tools  used  across  the  Warfare  Centers  are 
also  measures  of  effectiveness.  Baseline  costs  can  be  considered  lagging  indicators  of  future 
activity.  Trends  seen,  as  these  types  of  data  are  captured,  can  become  lead  indicators  for  further 
success. 

4.2  Success 

Success  will  be  at  hand  only  when  an  independent  but  fully  integrated  team  is  responsible  for 
distributing  test  requirements,  ascertaining  which  facilities  achieve  what  capability  level  and  are 
operating  together  as  a  single  entity.  Success  is  also  dependent  on  test  funding  as  a  single  stream 
managed  by  a  single  set  of  leaders — and  on  a  single  team  managing,  distributing,  and  reporting 
test  process  results  that  have  been  system  engineered  to  achieve  desired  outcomes.  When  the 
aligned  approach  is  institutionalized  across  the  Surface  Navy  test  community,  then  success  will 
have  occurred  and  new  problems  will  take  the  better  part  of  our  attention  and  our  time.  Until 
then,  pay  now,  pay  more,  and  then  pay  again  later. 


Capability  is  defined  as  processes,  people,  and  facilities  needed  to  effect  positive  outcomes. 
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